Finding content-bearing terms using term similarities

This paper explores the issue of using different co-occurrence similarities between terms for separating query terms that are useful for retrieval from those that are harmful. The hypothesis under examination is that useful terms tend to be more similar to each other than to other query terms. Preliminary experiments with similarities computed using first-order and second-order co-occurrence seem to confirm the hypothesis. Term similarities could then be used for determining which query terms are useful and best reflect the user's information need. A possible application would be to use this source of evidence for tuning the weights of the query terms. 1 I n t r o d u c t i o n Co-occurrence information, whether it is used for expanding automatically the original query (Qiu and Frei, 1993), for providing a list of candidate terms to the user in interactive query expansion, or for relaxing the independence assumption between query terms (van Rijsbergen, 1977), has been widely used in information retrieval. Nevertheless, the use of this information has often resulted in reduction of retrieval effectiveness (Smeaton and van Rijsbergen, 1983), a fact sometimes explained by the poor discriminating power of the relationships (Peat and Willet, 1991). It was not until recently that a more elaborated use of this information resulted in consistent improvement of retrieval effectiveness. Improvements came from a different computation of the relationships named "second-order co-occurrence" (Schutze and Pedersen, 1997), from an adequate combination with other sources of evidence such as relevance feedback (Xu and Croft, 1996), or from a more careful use of the similarities for expanding the query (Qiu and Frei, 1993). Indeed, interesting patterns relying in cooccurrence information may be discovered and, if used carefully, may enhance retrieval effectiveness. This paper explores the use of co-occurrence similarities between query terms for determining the subset of query terms which are good descriptors of the user's information need . Query terms can be divided into those that are useful for retrieval and those that are harmful, which will be named respectively "content" terms and "noisy" terms. The hypothesis under examination is that two content terms tend to be more similar to each other than would be two noisy terms, or a noisy and a content term. Intuitively, the query terms which reflect the user's information need are more likely to be found in relevant documents and should concern similar topic areas. Consequently, they should be found in similar contexts in the corpus. A similarity measures the degree to which two terms can be found in the same context, and should be higher for two content terms. We name this hypothesis the "Cluster Hypothesis for query terms", due to its correspondence with the Cluster Hypothesis of information retrieval which assumes that relevant documents "are more like one another than they are like nonrelevant documents" (van Rijsbergen and SparckJones, 1973, p.252). Our middle-term objective is to verify experimentally the hypothesis for different types of co-occurrences, different measures of similarity and different collections. If a higher similarity between content terms is indeed observed, this pattern could be used for tuning the weights of query terms in the absence of relevance feedback information, by increasing the weights of the terms which appear to be content terms, and inversely for noisy terms. Next section is about the verification of the hypothesis on the CACM collection (3204 documents, 50 queries).